{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enviroment: \n",
    "Open AI gym [MountainCarContinuous-v0](https://github.com/openai/gym/wiki/MountainCarContinuous-v0)\n",
    "\n",
    "### Observation\n",
    "Type: Box(2)\n",
    "\n",
    "| Num  | Observation | Min   | Max  |\n",
    "| ---- | ----------- | ----- | ---- |\n",
    "| 0    | position    | -1.2  | 0.6  |\n",
    "| 1    | velocity    | -0.07 | 0.07 |\n",
    "\n",
    "### Actions\n",
    "\n",
    "Type: Discrete(3)\n",
    "\n",
    "| Num  | Action     |\n",
    "| ---- | ---------- |\n",
    "| 0    | Push car to the left (negative value) or to the right (positive value |\n",
    "\n",
    "### Reward\n",
    "\n",
    "Reward is 100 for reaching the target of the hill on the right hand side, minus the squared sum of actions from start to goal.\n",
    "\n",
    "This reward function raises an exploration challenge, because if the agent does not reach the target soon enough, it will figure out that it is better not to move, and won't find the target anymore.\n",
    "\n",
    "Note that this reward is unusual with respect to most published work, where the goal was to reach the target as fast as possible, hence favouring a bang-bang strategy.\n",
    "\n",
    "### Starting State\n",
    "\n",
    "Position equal to 0.5. A constraint on velocity might be added in a more challenging version.\n",
    "\n",
    "Adding a maximum number of steps might be a good idea.\n",
    "\n",
    "### Episode Termination\n",
    "\n",
    "The episode ends when you reach 0.5 position, or if 200 iterations are reached.\n",
    "\n",
    "### Solved Requirements\n",
    "Get a reward over 90. This value might be tuned.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. gym enviroment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.49057207,  0.        ])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "env = gym.make(\"MountainCarContinuous-v0\")\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Q Table setup\n",
    "\n",
    "As action is an continue value \n",
    "we should build an discrete action spaces for the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEARNING_RATE = 0.5\n",
    "DISCOUNT = 0.95\n",
    "EPISODES = 10000\n",
    "SHOW_EVERY = 200\n",
    "Q_TABLE_LEN = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "action_space = np.array(range(-10,11,4))/10.\n",
    "action_space = action_space.reshape(len(action_space),1) #to fit with the enviroment\n",
    "\n",
    "DISCRETE_OS_SIZE = [Q_TABLE_LEN] * len(env.observation_space.high)\n",
    "discrete_os_win_size = (env.observation_space.high - env.observation_space.low) / DISCRETE_OS_SIZE\n",
    "\n",
    "q_table = np.random.uniform(low=0, high=1,\n",
    "                            size=(DISCRETE_OS_SIZE + [len(action_space)]))\n",
    "\n",
    "# q_table = np.zeros(DISCRETE_OS_SIZE + [len(action_space)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((200, 200, 6), (6, 1))"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_table.shape , action_space.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decay epsilon "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon = 1  # not a constant, qoing to be decayed\n",
    "START_EPSILON_DECAYING = 1\n",
    "END_EPSILON_DECAYING = EPISODES//2\n",
    "epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Help functions \n",
    "\n",
    "**action_indx** and **action** value correspond to **Q[state][action_indx]** and **action**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_discrete_state (state):\n",
    "    discrete_state = (state - env.observation_space.low) // discrete_os_win_size\n",
    "    return tuple(discrete_state.astype(int))\n",
    "\n",
    "def take_epilon_greedy_action(state, epsilon):\n",
    "    discrete_state = get_discrete_state(state)\n",
    "    if np.random.random() < epsilon:\n",
    "        action_indx = np.random.randint(0,len(action_space))\n",
    "    else:\n",
    "        action_indx = np.argmax(q_table[discrete_state])\n",
    "    return action_indx, action_space[action_indx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Rewards Recorder setup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_rewards = []\n",
    "aggr_ep_rewards = {'ep':[],'avg':[],'min':[],'max':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train the Agent \n",
    "Different from \"MountainCar-v0\" action_indx and action in action_space are added"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 0\n",
      "episode: 200\n",
      "episode: 400\n",
      "episode: 600\n",
      "episode: 800\n",
      "episode: 1000\n",
      "episode: 1200\n",
      "episode: 1400\n",
      "episode: 1600\n",
      "episode: 1800\n",
      "episode: 2000\n",
      "episode: 2200\n",
      "episode: 2400\n",
      "episode: 2600\n",
      "episode: 2800\n",
      "episode: 3000\n",
      "episode: 3200\n",
      "episode: 3400\n",
      "episode: 3600\n",
      "episode: 3800\n",
      "episode: 4000\n",
      "episode: 4200\n",
      "episode: 4400\n",
      "episode: 4600\n",
      "episode: 4800\n",
      "episode: 5000\n",
      "episode: 5200\n",
      "episode: 5400\n",
      "episode: 5600\n",
      "episode: 5800\n",
      "episode: 6000\n",
      "episode: 6200\n",
      "episode: 6400\n",
      "episode: 6600\n",
      "episode: 6800\n",
      "episode: 7000\n",
      "episode: 7200\n",
      "episode: 7400\n",
      "episode: 7600\n",
      "episode: 7800\n",
      "episode: 8000\n",
      "episode: 8200\n",
      "episode: 8400\n",
      "episode: 8600\n",
      "episode: 8800\n",
      "episode: 9000\n",
      "episode: 9200\n",
      "episode: 9400\n",
      "episode: 9600\n",
      "episode: 9800\n"
     ]
    }
   ],
   "source": [
    "for episode in range(EPISODES):\n",
    "    # initiate reward every episode\n",
    "    ep_reward = 0\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        print(\"episode: {}\".format(episode))\n",
    "        render = True\n",
    "    else:\n",
    "        render = False\n",
    "\n",
    "    state = env.reset()\n",
    "    done = False\n",
    "    while not done:\n",
    "        action_indx, action = take_epilon_greedy_action(state, epsilon)\n",
    "\n",
    "        next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "        ep_reward += reward\n",
    "\n",
    "        # if render:\n",
    "        #     env.render()\n",
    "\n",
    "        if not done:\n",
    "\n",
    "            td_target = reward + DISCOUNT * np.max(q_table[get_discrete_state(next_state)])\n",
    "\n",
    "            q_table[get_discrete_state(state)][action_indx] += LEARNING_RATE * (td_target - q_table[get_discrete_state(state)][action_indx])\n",
    "\n",
    "        elif next_state[0] >= 0.5:\n",
    "            # print(\"I made it on episode: {} Reward: {}\".format(episode,reward))\n",
    "            q_table[get_discrete_state(state)][action_indx] = 0\n",
    "\n",
    "\n",
    "        state = next_state\n",
    "\n",
    "    # Decaying is being done every episode if episode number is within decaying range\n",
    "    if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:\n",
    "        epsilon -= epsilon_decay_value\n",
    "\n",
    "    # recoard aggrated rewards on each epsoide\n",
    "    ep_rewards.append(ep_reward)\n",
    "\n",
    "    # every SHOW_EVERY calculate average rewords\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        avg_reward = sum(ep_rewards[-SHOW_EVERY:]) / len(ep_rewards[-SHOW_EVERY:])\n",
    "        aggr_ep_rewards['ep'].append(episode)\n",
    "        aggr_ep_rewards['avg'].append(avg_reward)\n",
    "        aggr_ep_rewards['min'].append(min(ep_rewards[-SHOW_EVERY:]))\n",
    "        aggr_ep_rewards['max'].append(max(ep_rewards[-SHOW_EVERY:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Rewards')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEKCAYAAADuEgmxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd4XMXVwOHfbJFWvcuSVWzJvTe5G2MwEMwHmJoAAUw1NbQQWkJIAiG0QAgQwPRiOhhMBxuMTTHGvQsXyVaz1fuutGW+P+5KbrKttlqV8+a5z7a7e0dOsmfnzMwZpbVGCCGEaA6TvxsghBCi65CgIYQQotkkaAghhGg2CRpCCCGaTYKGEEKIZpOgIYQQotkkaAghhGg2CRpCCCGaTYKGEEKIZrP4uwHtITY2Vvft29ffzRBCiC5l1apVxVrruJa8p1sEjb59+7Jy5Up/N0MIIboUpdSulr7H5+kppdSLSqlCpdTG/Z6LVkp9rZTa5r2N8j6vlFL/VUptV0qtV0qN9XX7hBBCNF9HjGm8DJx80HN3AIu11gOAxd7HALOAAd5jLvB0B7RPCCFEM/k8aGitlwKlBz09G3jFe/8V4Iz9nn9VG5YDkUqpRF+3UQghRPP4a0yjl9a6AEBrXaCUivc+nwTk7Hderve5gpZewOl0kpubi8PhaHNjOyubzUZycjJWq9XfTRFC9BCdbSBcNfFckxt+KKXmYqSwSE1NPeT13NxcwsLC6Nu3L0o19bFdm9aakpIScnNzSUtL83dzhBA9hL/WaextSDt5bwu9z+cCKfudlwzkN/UBWut5WusMrXVGXNyhM8YcDgcxMTHdMmAAKKWIiYnp1j0pIUTn46+gsRCY470/B/hov+cv9s6imgRUNKSxWqO7BowG3f3vE0J0Pj5PTyml3gRmALFKqVzgHuAB4B2l1OXAbuBc7+mfAacA24Fa4FJft08IIRrUu+vZW7MXs8mM1WQ1DrNxazFZMKnW/862u+xkVWSxo3wHlfWVRAVGEWmLJCowiiibcQSYAqisr6TUUUqJvYQSRwmljlLKHeV48KAa/qOMW5MyMSpuFBMSJ7Tjv8KR+TxoaK3PP8xLM5s4VwPX+bZFQoiupLK+kqW5S/lm9zesK1xHfHA8fSP6khaRRt/wvvSN6Euf8D6YlAmHy2EcbuO2zl2H1WQlxBpCqDWUkIAQrCZj4ojL42JH+Q42l2xmY/FGNpVs4teyX3F6nE22Q6GIskURHxxPXFAcccFxxm1QHBaTBQ8etNa4tRuP9uDRHgprC9lRvoOdFTvJr85HNz1E28iszLi1u0X/PpcNv6x7BQ0hRNentaasrozdlbspc5SRHplOSlhKm355H8nemr18m/Mt3+z+hl/2/IJLu4gLimNC4gRK7CX8sucXPtn5Sas+22a2EWINocZZg8NtjAmGWkMZFjOMi4ZeRFpEGlprnB6ncbiNW4fbQYm9hCJ7EUW1RWwp3UKpoxSP9hz2WgGmANIi0hgZN5Iz+p9Bv8h+9IvoR6QtkvK6csocZZQ7yimtM3oTta5aogKjiA6KJsYWQ0xQDNG2aKICozCbzGit0Wi01njwgO74NLUEDR8744wzyMnJweFwcOONN+J2u8nKyuKhhx4C4OWXX2bVqlU88cQT3HvvvcyfP5+UlBRiY2MZN24ct956q5//AtGZVNRVsHLPSnZU7CA+OJ6k0CR6h/amV3AvLKZ9/3d2eVzsqdlDbnUueVV55FXnYTVbjS+i/b6MYoJiACivK6e8rpwKR0Xj/SJ7Ebsrd5NTlUNOVQ7VzuoD2hJiDWFQ1CAGRw9mcPRg+kf2p6K+ovH8nKoccqtyyavOo85dhwkTJnXgoVB4MH6Va62NW3Tjr/2+4X25eNjFzEydyfDY4QcEqVpnLbsqd5Fdmc3uyt0A2Cw2gixBBJoDsVls2Mw2XB4X1c5qqp3V1DhrqK437geaAxkWO4zhMcNJDU9tVQB0eVyUOcpwa/cBf5NZmVFKEWoNxWwyN/neaFs0RLTseg1pKRSYafpzfa1HBI2/f7yJzfmV7fqZQ3uHc89pw4563osvvkh0dDR2u53x48ezePFipk6d2hg03n77bf785z+zcuVK3n//fdasWYPL5WLs2LGMGzeuXdssup56dz1rC9eyvGA5ywuWs6lkU5O/bM3KTEJIArFBsRTbi9lTs+eANEdr0h4WZSEpLInksGRGx48mNSyV1PBUIgIj2FG+gy0lW9haupUF2xdgd9kPeK/NbCM5LJnksGQmJU4i2BrcmLLZ/9Bo48vWG1BQYMJERGAE05Onkx6Rfthf0sHWYIbEDGFIzJAW/V3tyWKyEBfconp/XV6PCBr+9N///pcFCxYAkJOTQ1ZWFunp6SxfvpwBAwaQmZnJ1KlTefzxx5k9ezZBQUEAnHbaaf5stvBq+DVbbC+m2F5MiaOk8X65o5w6d90BR727HqfHic1iI8QSQog1hGBrMCFW435cUByJIYkkhiaSEJJAYkgiYQFhuDwu8qrzyKrIajx2VuwkszQTh9uBWZkZETuCuSPnMjlxMoOjB1NiLyGvJq+xJ5Ffk09RbREj40ZyStoppISlkBRqfOn3Cu6FR3soqys7YIC1xF4CQGRgJBGBEUQGRjbejwiMOKD3sr9RcaOMYj+A2+Nmd9VudlbsJNoWTXJoMrFBsTK7r5vqEUGjOT0CX1iyZAmLFi3ip59+Ijg4mBkzZuBwOPjd737HO++8w+DBgznzzDNRSmHMARD+prVmV+UuluUtY1nuMlbuXXnIwGiYNYyYoBgiAyOxWWxEBEYQYA4g0BxIoDkQi8mC3WWn1llLjbOGGlcNxfZiqp3VFNcW49KuAz4v1BpKnbvugOvE2GJIi0jj7IFnMylxEhm9MggNCD3gfcHWYFLCU6CZhXbMmIkPjic+OP7oJ7eA2WQmLSKNtAhZZNoT9Iig4S8VFRVERUURHBzM1q1bWb58OQBnnXUW//znP+nTpw8PPvggANOmTeOqq67izjvvxOVy8emnn3LllVf6s/k9RrmjnLVFa/kx/0eW5S4jtzoXgPSIdC4YfAGj4kc1zpaJscVgs9hafS23x02Jo4SCmgIKqguM25oCbGYbaRFppEem0ze8LxGBLUx2C9FBJGj40Mknn8wzzzzDyJEjGTRoEJMmTQIgKiqKoUOHsnnzZiZMMKbKjR8/ntNPP51Ro0bRp08fMjIyiIiQL4725tEesiqyWFu4lrVFa1lbuJbsymzAyMNPTJzIJcMuYVryNJJCk9r9+mbTvl/7o+JGtfvnC+FrqjukRTIyMvTBmzBt2bKFIUP8N0DWGtXV1YSGhlJbW8v06dOZN28eY8ceeUuRrvR3Ot1OsiqNfH1FXYWRuvEeDTNb0iLSmJE8g2Gxw1o1m0VrzdbSrazYs4JiezFljjLK6rzTGh2llDhKGgdtIwMjGR03mlHxoxgdN5oRcSMINAe2958tRKellFqltc5oyXukp9GJzJ07l82bN+NwOJgzZ85RA0Zn5va4WZa3jK2lW9levp3tZdvZVbnrkHy+QjUuurKZbSzevZh56+cRGxTL9OTpHJt8bOPsm8PxaA/ri9azaNciFu1eRF51HmDMkY+yRRFtiyYyMJLksGSibdEMih7E6LjR9AnvI4O1QrSQBI1O5I033vB3E9rNY6se45XNxpYpyaHJ9I/qz/GpxzMgagDpEelE26IJsYYQZAk64Iu73FHOsrxlfJf7HV9lf8UH2z4gwBRA/6j+hAWEER4QTqg1lLCAMEIDQilzlLF492IKawuxmCxMTJzIlSOu5NiUY4mxdd+ClUL4iwQN0e7WFq7l1c2vctaAs7h9/O1H7CUcLNIWyWn9TuO0fqfh9DhZvXc1S3KWkF2ZTVV9FcW1xVTVV1HlrMLushNoDmRq76mcMPYEjk05lvCAcB/+ZUIICRqiXTlcDu7+4W4SQhK4bfxtLQoYB7OarExMnMjExIlNvu70OEGD1SybUAnRUSRoiHb15Jonya7MZt6J8wixhvj0Wg2F54QQHcdf+2mIbqghLXXuwHOZ3Huyv5sjhPABCRp+tnDhQh544IEDnuuK06D3T0vdMu4WfzdHCOEjkp7ys9NPP53TTz+98bHT42Rn+U4SQhK61Krgp9Y+1ZiWOrjchRCi+5Cg4UPZ2dmcfPLJTJs2jeXLlzNq1CguvfRS7rnnHgoLC5k/fz6bN29m5cqVPPnkk1xyySWYg8ysWrWK0qJSHn34Uc455xx//xlH1ZCWOmfgOZKWEsLroS+2siGvgpToYFKigkmJDiIlKpjU6GAig61ddjp4zwgan98Beza072cmjIBZDxz1tO3bt/Puu+8yb948xo8fzxtvvMH333/PwoULuf/++znjjDMaz/VoDwX5Bbz2yWvs2r6Lmy6+qdMEjUW7FrGuaB1xQXGNZTDiguOICIzg7h/upldwL/447o/+bqboxn7dW8VVr63i+TkZ9Ivr3L3ZFVml/G/JDlKjg9mYV0FZ7YFFL6f0i2H+FRO7ZODwa9BQSt0MXAFoYAPGnuCJwFtANLAauEhrXe+3RrZRWloaI0aMAGDYsGHMnDkTpRQjRowgOzv7gHMdLgczZ82kV2gvTANN7N27t1nXsLvs1LnqiLRFNuv8YnsxX2V/xWn9TiMsIOyo5y/LXcYtS27BpEyH3ZPh2ROflbSU8Kll24rJKq7h0a9/5akLOm+1BI9Hc+8nm0mMsPHlTdMJCjBTXecip7SW3aW1LNtWxOvLd7Miq5SJ6TH+bm6L+S1oKKWSgBuAoVpru1LqHeA84BTgMa31W0qpZ4DLgafbdLFm9Ah8JTBwXy0jk8nU+NhkMuFy7Sup0bAXQ1RoFJGBkRTVFjV7QPy+5ffxWdZn/G7Q77hyxJWNu7EdzOl28sbWN3hm3TNUO6v5etfXPHPiM0est5RTlcMdy+5gYNRAXjvlNerd9RTWFlJUW8Te2r0U2YvoFdyLKb2nNKutQrTWlgJjI7VP1xdw/XGVDEnsnAs5P1iTx4a8Cv7zu9EEBRi764UGWhiSGM6QxHCmD4jjo7X5vPVLTrODxtqcckYmRWAy+b9n4u/ZUxYgSCllAYKBAuB44D3v668AZxzmvd1KUW0RKAgLCMNqsmI1WY+6CT0YM60W715MfFA8b219i1kfzOKptU9RXX/g1pxLc5dy1sKzeGTlI4yOH82tGbeycu9KbvvuNlweV5OfbXfZuWXJLWg0j814jCBLEBGBEQyIGsCUpCmcOeBM5o6cy+z+s9vl30CII9lSUMmolEjCbBYe+/pXfzenSTV1Lh76YiujUyI5fVTvJs8JCjBzxugkPt1QQHnt0ZMo32YWcsZTP/Da8l3t3dxW8VvQ0FrnAY8AuzGCRQWwCijXurGqXS7Q/vWpOxmXx0VFXUXjBj5KqcaFcUfrbdR76qlx1nD7hNtZMHsBxyQdwzPrnmHWB7N4ddOrZJZmcu2ia7lu8XUAPDXzKZ4+4WnmDJvDHRPu4Jucb7hv+X2HXEdrzb0/3UtmaSYPHvOgsdmPEH7icnvYtreaiWnRXDEtna8272VDboW/m3WIZ7/bQWFVHXefOvSIvYLzJqRQ7/KwYE3eET9Pa82T32wHYN7SnTjdh27129H8mZ6KAmYDaUA58C4wq4lTm/zWVErNBeYCpKam+qiVbdO3b182btzY+Pjll19u8rWZZ8+kur6a+a/Ob9xeM8QaworsFdS564646Y/D5SDAFNBYCfbfM/7NpuJNPL76cR5e+TBg7Ax3a8atXDD4ggNKbvx+yO8psZfw3IbniLZFc8PYGxpfeyvzLT7e+THXjr6WY5KPaZd/DyFaa2dxDfVuD0MSw5g5pBcv/pDFo19n8tKlE/zdtEZ55XaeXbqT00f1ZlyfqCOeO6x3BKOSI3hrRQ6XTOl72AHxn7NKWbWrjJmD41m8tZDPNhQwe7R/f0f7Mz11ApCltS7SWjuBD4ApQKQ3XQWQDOQ39Wat9TytdYbWOiMurutu7G532amsqyQmKOaA/ZgbajbVOGsO+16tNXXuOiYmTjygxtOw2GHMO2kez530HDeOvZGPz/yYOcPmNFmj6Q9j/sDZA87muQ3P8frm1wFjCu1DKx7i2ORjuWrkVe31pwrRag3jGUMSwwm3WZk7PZ1vM4tYvbvMzy3b56EvtgJw+6zBzTr/vAmpZO6tYvXu8sOe89S324kNDeTJC8YyID6UZ77b6ffFv/4MGruBSUqpYGWE2ZnAZuBboGGe6RzgIz+1r0MU1RZhVmZibAcOiAWYA7CarUcMGnXuOlweFzNSZjT5+qTESVwx4gpig2IP+xlKKe6edDcnpJ7Ag788yGubX+OWJbeQGJrI/cfc36qNkIRob5sLKrGaVeNU20um9CUmJKDTjG2s3l3GR2vzmTs9naTIoGa957RRvQkOMPPWit1Nvr4+t5xl24q54pg0ggLMzJ2ezpaCSr77tag9m95i/hzT+BljwHs1xnRbEzAPuB24RSm1HYgBXvBXG32t1llLVX0VMUExmE3mQ14PsYZQ46w57C+LqvoqAKYnT29TO8wmMw9Mf4DxCeN56JeHqKqv4rEZj0mZcdFpbC2oon98GFaz8ZUVEmjh6mP7sWxbMSuySv3aNq01//h4M/FhgVx9bL9mvy800MLs0b35ZH0BlQ7nIa//79sdhNss/H6ikX6fPTqJxAgbz3y3o93a3hp+/Rmptb5Haz1Yaz1ca32R1rpOa71Taz1Ba91fa32u1rrOn230pcLaQswmM9G26CZfD7GG4NEeHG5Hk69XO6uxmqwkhCS0uS2B5kAeP+5xTu57Mg9Of5BB0YPa/JlCtJctBZUMSTxwTdGFk/oQFxbIv7/K9GvKZuG6fNbmlPOn3wwiJLBlw8TnjU/F7nTz0doDs/Db9lbxxaY9XDKlL2E2I60cYDFx+bQ0lu8sZW3O4VNavia5hw7m8rgodZSSXZFNjbOG2KDYJnsZACEWYwZVUykql8dFrbP2iIPkLRUWEMbDxz7M8anHt9tnCtFWJdV1FFbVMfSgdRlBAWaum9GPn7NK+XFHiV/aVl5bzwOfb2V4Ujhnj01u8ftHJkcwJDH8kBTV00t2EGQ1c8nUtAOeP29CKuE2C8/6sbchQaMD7B8oMkszKaguwOlxEhccd9heBhibCwWYA6h11h7yWsM6DJu5/YKGEJ3RlgIjDdvUYr7zJqSSGGHzS2/D7dH84c01FFfXce/s4a1aeKeU4oIJKWzKr2ycQpxTWstH6/K5YGIq0SEBB5wfGmjhosl9+GLTHnYWVTf1kT4nQcPHimqLDggUsUGxpEem0z+yP/HB8UcdaD7cuEaVswqLyYLF3DPKh4mea+seY+bU4IRDS97YrGauO64/q3eX883Wwg5t18NfZrJsWzH/mD2cMalHnmJ7JLPHJGGzmnjD29t4dukOzEpx5THpTZ5/yZQ0rGYTzy3b2eprtoUEDR8qthdTWFtIeGB4Y6DoFdKLIEtQswuVNY5ruPaNa3i0h+r6asICwlD4v6yAEL60uaCS+LBAYkKbLnfz24wU0mND+NvHm7DXN10brb19sj6fZ77bwQUTUzl/QtvWiYXbrPzfiN4sXJtHVnEN76zM5exxySRENJ1FiAsL5Nxxyby/Ko/CyqbHO31JgoaPlDpKWb1lNbOnzOZvN/2N8aPHc+GFF7Jo0SKmTp3KgAEDWLFiBStWrGDKlCmMGTOGKVOmkJmZCcCjjz7KZZddRrA1mF83/8rY0WOprTXSVLXOWjzaQ6hVCgSK7m9LQdUR60wFWEz888wR5JTa+c9i30/B3ZxfyZ/eXU9Gnyj+dtqwdvnMCyamUFPv5tKXVuBye7j62KZ7GQ3mTk/H5fHw4g/Z7XL9lugRuY0HVzzI1tKt7fqZg6MHc/uE25t8rdxRTkF1ASHWELJ3ZrPg/QUMGzasydLor776KkuXLsVisbBo0SLuuusu3n//fW666SZmzJjBJx99wl//8Vfue+w+goONBXxVziqUUlJVVnR79S4P2wurOHbgkRfwTu4Xw7njknl+WRazRyUxtLdvpouX1dRz1esrCQ+y8L8LxxJgaZ/f3WNToxgQH8q2wmpmj+5Nn5iQI57fJyaEWSMSmb98F9cd169xhlVHkJ5GO6uoqyCvOo8QawiJoYmNpdFNJlOTpdErKio499xzGT58ODfffDObNm0CjCq4L7/8MhdddBFTpk1h6LiheLQHrTVV9VWEWENk4Z3o9nYWV+N060Om2zblrlOGEBFk5c4FG3B72n9Q3OX28Ic317C3oo5nLhxHfFj7TUJRSnHR5D6YTYprZjRvrcfV0/tRVefijZ+bXhzoKz2ip3G4HkFrON1OPHiwmg6N7FX1VeRV5RFsDSYlLIWcspyjlka/++67Oe6441iwYAHZ2dnMmDGj8fxt27YRGhpKyd6SxnENkzLhdDuPuMpbiO5i//IhRxMVEsDdpw7h5rfX8fryXcyZ0rdd2/LQl5l8v72Yh84Z2aaB78O5cGIfThzai8SI5q0oH5EcwWVT0xjcwSXi5adqM3m0h8LaQraVb2N72Xa2lGwhszSTneU7yanKoaC6gJyqHGwWG6lhqYdde3GwiooKkpKMAmT7FzSsqKjgxhtvZOnSpVSUVfDVwq+ocdY0rgIPsx79l5cQXd2WgioCLCbSY4+crmlwxugkjhkQy8NfZrKnov0GiRdv2cu8pTu5eHIffpvhm4rPJpNqdsBo8NfThh41ddfeJGg0g91lJ6sii6LaIsIDwkkKTSIuOI7QgFBMyoTD5aCsrgyb2UZqePMDBsBtt93GnXfeydSpU3G79838uPnmm7n22msZOHAgL774Iv+57z/syt9FtbMam8XWZPFBIbqbLQWVDOwVisXcvK8qpRT3nTEcp9vDPQs3Hv0NzVDlcPKXDzcyqFcYf/m/oe3ymV2Z8nfFxPaQkZGhV65cecBzW7ZsYciQIW36XI/2UFRbRLG9GIvJQmJIIuGBTXcFG/4dfbXnb0F1AWV1ZWitiQs29umG9vk7heisMu5bxIxBcTxy7qgWve+pb7fz8JeZzLtoHCcNa1uZnb9+tJHXlu/ig2um+CQt5U9KqVVa64yWvEd6GodR66xlZ/lOiu3FRAZG0i+y32EDBhjBwpebxIdYQxoDk6SmRE9QVFVHcXVdq7Z1nTs9nUG9wrhn4Saq65rembI5VmaX8tryXVwypW+3Cxit1SMGwlvK5XGRXZmNRVlIDU8lLMD/X9IN+2VYTJZ2rTclRGe1bxC85f//s5pN3H/WCM555kemPvAN4UEWgqxmgqxmbFYzQQFmRqdEcv1x/Q+b+qpzubn9/fX0jgji1pOkgGeDbh00tNat+vXv8rjQWtMrtFenCBhgBIuwgDBsFlvj39QdUotCHE5j0Eho3eygcX2iePy8MSzfWYKj3o3d6T3q3RRW1vGfRdtYs7ucJy4YQ3gT6xye+nYHO4pqePnS8S2uXtudddt/CZvNRklJCTExMS0OHA1fxp1tHURq+L5yBVprSkpKsNmk1yG6p617qkgItxF1UNG+ljh9VG9OH9W7ydfeXLGbuz/cyDlP/8gLc8aTEr1v98vMPVU8vWQ7Z45JYsag+FZfvzvqtkEjOTmZ3NxciopavstVnbuOEnsJziAngeam6910BjabjeTklpdjFqK9ldfW8/YvOZybkXJIZdbWamoPjfZ0/oRU+kQHc/XrqzjjqR+Yd/E4xvWJxu3R3P7+esJsVu4+VWZLHazbBg2r1UpaWtrRT2zCstxl3LTqJl4/5XWGxMnMJCGO5Pttxdz67jr2VDrYU+ngnnaox1TncrO9sJrjB/v2V/6U/rEsuG4ql7/8C+fP+5mHzhlJWW09a3PKefy80e0WALuTzpV/6STsLjsAQZaWLbQRoidxON38/eNNXPjCz4QEmpmcHsO7K3OpamLr0pbaXliNy6NbNXOqpfrFhbLg2qmMSY3kprfX8q/PtjJjUNxh01o9nV+DhlIqUin1nlJqq1Jqi1JqslIqWin1tVJqm/e2w+e5NWyvGmSWoCFEUzblV3DaE9/z0g/ZzJnch0/+cAy3zxpMdZ2L91bltvnztzZuvNQxE1GiQgJ47fKJnDc+hfAgC/edMdynU+i7Mn+npx4HvtBan6OUCgCCgbuAxVrrB5RSdwB3AO1XPKoZ7E5vT8MqQUOI/Xk8mmeX7uTRrzOJCg7glcsmNJaxGJ0SyZjUSF75MZs5k/u2aie7BlsKKgm0mOh7lGqv7SnAYuKBs0fi8eg2tb2781tPQykVDkwHXgDQWtdrrcuB2cAr3tNeAc7o6LY1pKdkK1UhDvTKT9k8+MVWThjSiy9vmn5I3aNLp6aRXVLLkl/btovelj2VDEoIa3b5kPYkAePI/JmeSgeKgJeUUmuUUs8rpUKAXlrrAgDvbYfPd7O7vUFDFtEJ0ai23sVT325ncnoM//v92Canws4ankCv8EBeasPmQFprY+OlVq7PEL7lz6BhAcYCT2utxwA1GKmoZlFKzVVKrVRKrWzNtNojsbvsWE1WLCZ/Z++EaLsqh5Ob3lrDN1v3tulzXvlxF8XV9dz6m4GHzfdbzSYumtSHZduK2ba3qlXXKaqqo7SmnsEdNJ4hWsafQSMXyNVa/+x9/B5GENmrlEoE8N422c/VWs/TWmdorTPi4tq3NLDD5ZCZU6JbqK13celLv/Dh2nz++M46SqrrWvU5lQ4nz3y3g+MGxTGuT/QRzz1/QioBFhMv/ZjdqmttbsEeGqLj+S1oaK33ADlKqYaiLjOBzcBCYI73uTnARx3dNrvLLkFDdHkOp5srX13J6t1l/Ok3g6iuc3Hfp1ta9Vkvfp9Fhd3JLScevQZTTGggZ4zuzQercymvrW9Rez9dX8Dji7cBrS8fInzL3/mXPwDzvTOndgKXYgSyd5RSlwO7gXM7ulHS0xBdXb3Lw7XzV/PD9hL+fe4ozh6XTJ3Lw38Xb+OMMUkt2rinrKaeF5ZlcfKwBEYkRzTrPZdOTeOdlbm8/UsOVx17+O1Ltdasy63gvVU5fLyugAq7k8QIG387bSgRwbJnTGfk16ChtV4LNFXLfWZHt2V/0tMQvuD2aMwdMDPH5fZw41tr+GZHETZ6AAAgAElEQVRrIf88czhnjzNKzVx3XD8+XZ/PXR9s4Kubpze7CN+zS3dSXe/ilpMGNrsNQxLDmZQezas/7eLyaWmHzIKqd3l4+5fdvPLTLrYXVhNoMXHy8ATOGZfMlH6xHfLvJFpHVoQ3we6yy8wp0a6+2FjAmH98xd8WbsLj8V11Yo9H86f31vP5xj3cfepQfj+xT+NrgRYzD5w9krxyO49+/WuzPq+wysHLP2Yxe1RvBvZq2cD0pVPTyCu38/XmfQPwWms+21DASY99x90fbSI00MK/zhrBL385gcfPG8MxA+IkYHRy/k5PdUoOl4PQgFB/N0N0A26P5pGvMnl6yQ4Swm28/GM2Dqeb+88c0e7rAbTW/PnDDSxYk8etJw3k8mmH1l4b3zeaCyel8tIPWZw+qjejUiKP+JlPL9mB06258YTm9zIanDCkF8lRQbz0QzazRiSyIquU+z/bwtqccgb1CuOlS8czY2CcrLzuYqSn0YRaV60s7BNtVlZTzyUvreDpJTu4YGIq3902g+uP689bv+Twp/fW427nHscT32znzRU5XDujH9cfP+Cw59128mDiwgK5/f31ON2ew56XX25n/vLdnDM2mbTYlq/MNpsUcyb3ZUV2KRc8t5zfPvsTeyocPHTOSD678RiOGxQvAaMLkqDRBIfLISVERJtszKvg1Ce+5+edpTx49gjuP3MEgRYzt/5mELecOJD3V+dyyztrcR3hS7slPl1fwKNf/8pZY5P402+OPMMp3Gbl3tnD2bqniueW7TzseU9+ux2N5g8z+7e6Xb8dn0JooIUNuRXcdvIglvxpBr/NSJEUVBcm6akmyEC4aIv3V+Vy14INRIcE8O7Vkw9JAd0wcwBWs4kHv9iK0+3h8fPGYG1DuYz1ueX88d21ZPSJ4l9njWjWr/eThiUwa3gC/1m0jekD4ogLC8Tp9uD2aJxuTXF1He/8ksMFE1NJjgo+6ucdTkSQlc9vPIYwm4XIYCkz3h1I0GiCw+2Q9JRolaW/FvHHd9cxKT2aJy8YS2xo05t4XTOjH1az4r5Pt+B0r+bJC8YQaDG3+Hp7Khxc+epKYkICeeaicS36jL+fPozvtxdz6hPfN/m6zWriuuNa38tosP+OeKLrk6BxEK219DREqz3znTHg/eplEwmwHLn3cMUx6QRaTNz90SZm/WcZZ49LZvbo3s3+ZW+vNxbvVTtcvH/tlMMGqMOJD7fx7tWTWZFVitmksJpMWMzKuG82MSQxnF7h8uNJHEiCxkHqPfV4tEeChmixTfkV/LijhDtmDT5qwGhw0eS+xIfbeOH7LB7+MpOHv8xkYlo0Z41NYtaIRMJtTS9w83g0t767jo35FTx/cQaDW7l6enBCeKvfK3omCRoHcbi8GzBJ0BAt9PyyLEICzJw/IbVF7/vNsAR+MyyBnNJaPlyTx4I1edz+/gb++tEmJqXHkBYbQt+YYPrEhJAaE0xyVBD/+3YHn24o4M+nDGHmkF4++ouEOJQEjYM07qUhi/tECxRU2Pl4XT4XT+5LRFDryl+kRAfzh5kDuP74/qzLrWDB6lx+zirll+xSauvdjecpBVrDbzOSueKYQ9diCOFLEjQOIvuDi9Z4+cdsPFpz6dS+bf4spRSjUyIZ7Z11pbWmuLqe3aU1ZBfXsqu0FqtJcdWx/WSdg+hwEjQOIkFDtFR1nYs3ft7NrBGJPpkppJQiLiyQuLDAo5YlF8LXZHHfQRrGNCQ9JZrr7V9yqHK4uPKYdH83RQifk6BxkIaeRrBF5paLo3O5Pbz4fRbj+0Y1ppOE6M4kaBxEBsJFS3yxaQ955XaukF6G6CEkaBxExjREc2mteW5ZFmmxIZwg015FDyFB4yCNPQ0pIyKOYuWuMtbllHPZtDQpwCd6DAkaB2lc3CdVbsVRPLd0J1HBVs4Zm+zvpgjRYfweNJRSZqXUGqXUJ97HaUqpn5VS25RSb3v3D+8wkp4SzZFVXMPXW/Zy4aQ+BAW0vNCgEF2V34MGcCOwZb/HDwKPaa0HAGXA5R3ZGLvLjkVZsJpkU3vRtO+3FXPevJ+wWcxcNLnP0d8gRDfi16ChlEoG/g943vtYAccD73lPeQU4oyPb5HA7pJchmuRwuvnHx5u58IWfCQ208M5Vk4kPk7Ev0bP4e0X4f4DbgIYd62OAcq21y/s4F0jqyAbZXXaZbisOsTm/kpveXsOve6uZM7kPd8waImkp0SP5LWgopU4FCrXWq5RSMxqebuLUJjdSVkrNBeYCpKa2rKrokcheGmJ/bo/m+WU7eeSrTCKDA3j50vHMGBTv72YJ4Tf+7GlMBU5XSp0C2IBwjJ5HpFLK4u1tJAP5Tb1Zaz0PmAeQkZHRZGBpDQkaosGeCgc3v72Wn3aWcPKwBO4/awTRIbJlqejZ/DamobW+U2udrLXuC5wHfKO1/j3wLXCO97Q5wEcd2S6HyyHpKcGXm/Zw8uNLWZdbzkNnj+TpC8dKwBCCZvY0lFL9gFytdZ03lTQSeFVrXe6DNt0OvKWUug9YA7zgg2sclvQ0ejZ7vZv7Pt3M/J93MzwpnP+eN4b0uFB/N0uITqO56an3gQylVH+ML/GFwBvAKe3RCK31EmCJ9/5OYEJ7fG5r2F12omxR/rq88KMtBZXc8OYathVWM3d6OreeNKjZ27YK0VM0N2h4tNYupdSZwH+01k8opdb4smH+4nDJlNue6M0Vu7ln4SbCbVZevWwC0wfG+btJQnRKzQ0aTqXU+RhjDKd5n+uWq98kPdXzLN6ylzs/2MAxA2J57HejiQ0N9HeThOi0mtv3vhSYDPxTa52llEoDXvdds/xHeho9y66SGm56ey3Deofz3MUZEjCEOIpm9TS01puBG/Z7nAU84KtG+ZP0NHoOe72bq19fjUkpnrlwHDarLNYT4miOGDSUUhs4zOI6AK31yHZvkR85PU5c2iVl0XsArTV//nADW/dU8uIl432yt7cQ3dHRehqnem+v896+5r39PVDrkxb5kVS47Tnm/7ybD1bncdMJAzhOVngL0WxHDBpa610ASqmpWuup+710h1LqB+AfvmxcR7M7ZavXnmDN7jL+/vEmZgyK44bjB/i7OUJ0Kc0dCA9RSk1reKCUmgKE+KZJ/uNwezdgkp5Gt1VSXce181fTK9zGf343GpPsuCdEizR3yu1lwEtKqQiMMY4K73PdiqSnujeX28MNb62hpKaeD66ZQmSwlAURoqWOGjSUUiagv9Z6lFIqHFBa6wrfN63jNW71KkGjW/rX51v5YXsJD50zkuFJEf5ujhBd0lHTU1prD3C9935ldw0YALUuY2xfgkb3887KHF74PotLpvTltxkp/m6OEF1Wc8c0vlZK3aqUSlFKRTccPm2ZHzT0NGQgvHtZtauMvyzYyLT+sfzl/4b4uzlCdGktGdOAfVNvwRjbSG/f5viXjGl0PwUVdq56bRWJkTaevGAMFrMUIBSiLZq7IjzN1w3pDCRodC/2ejdzX12Fw+nmzSsnysC3EO2g2Tv3KaWGA0MxdtkDQGv9qi8a5S8yEN59aK257f31bMyv4PmLMxjQK+zobxJCHFVzN2G6B5iBETQ+A2YB3wPdKmg09DRkTKPr+9+SHXy8Lp/bTh7EzCG9/N0cIbqN5iZ4zwFmAnu01pcCo4BuVw7U7rJjUiYCTJLG6MpW7y7jka8yOX1Ub645tp+/myNEt9LcoGH3Tr11eddqFNLNBsFhX4VbpWSVcFf27spcgqxm/nXWCPnvUoh21tygsVIpFQk8B6wCVgMr2nJh7/Tdb5VSW5RSm5RSN3qfj1ZKfa2U2ua97bC9Vx1uh1S47eKcbg9fbCzghCG9CAls9pCdEKKZmhU0tNbXaq3LtdbPACcCc7xpqrZwAX/UWg8BJgHXKaWGAncAi7XWA4DF3scdQvbS6Pp+3FFCWa2TU0cm+rspQnRLzR0IfxVYBizTWm9tjwtrrQuAAu/9KqXUFiAJmI0x6A7wCrAEuL09rnk0dqddBsG7uE/W5RMWaOHYQbLHtxC+0Nz01MtAIvCEUmqHUur9hnRSe1BK9QXGAD8DvbwBpSGwdNhmBw63g2CLbMbTVdW7PHy5aQ8nDutFoEV24RPCF5q7uO8bpdR3wHjgOOBqYBjweFsboJQKBd4HbtJaVzZ34FIpNReYC5CamtrWZgBGekp6Gl3X99uLqHS4JDUlhA81q6ehlFoM/AD8DsgExmutB7f14kopK0bAmK+1/sD79F6lVKL39USMmVqH0FrP01pnaK0z4uLaJxXhcDlkTKML+2RdARFBVqb1l9SUEL7S3PTUeqAeGA6MBIYrpdr07aqMLsULwBat9aP7vbQQmOO9Pwf4qC3XaQkZCO+6HE43X23ey2+G9SLAIvWlhPCV5qanbobGVNKlwEtAAm1b4DcVuAjYoJRa633uLuAB4B2l1OXAbuDcNlyjRSQ91XV992sR1XUuTh3Z299NEaJba+7sqeuBY4BxwC7gRYzZVK2mtf4eONwAxsy2fHZrSU+j6/pkfQHRIQFM6Rfj76YI0a01d/VTEPAosEpr7fJhe/xKehpdk73ezeItezljTJKUPhfCx5q7uO9hwIqRTkIpFaeU6lbl0l0eF06PU3oaXdA3WwuprXfLrCkhOkBzZ0/dg7HA7k7vU1bgdV81yh8ayqLLOo2u55P1+cSFBTIxTVJTQvhac/vyZwKnAzUAWut8oFttUOBwe7d6ldpTXUp1nYtvthZyyvAEzCYpTiiErzU3aNRrrTXGFq8opUJ81yT/sDu9u/ZZJT3VlSzespc6l4dTR8msKSE6QnODxjtKqWeBSKXUlcAi4HnfNavj1bpqAelpdDUfrysgIdzGuNQOK4YsRI/W3HUajyilTgQqgUHAX7XWX/u0ZR2sIT0lA+FdR6XDydJfi7hwUh9MkpoSokM0e8MBb5D4GkApZVZK/V5rPd9nLetgstVr1/PBqlzq3R5OHSWzpoToKEdMTymlwpVSdyqlnlRKnaQM1wM7gd92TBM7hsye6jrcHs2/v8rkbx9vJqNPFGNSIv3dJCF6jKP1NF4DyoCfgCuAPwEBwGyt9dojvbGraehpSHqqcyuvrefGt9by3a9F/DYjmX/MHi5bugrRgY4WNNK11iMAlFLPA8VAqta6yuct62ANPQ1JT3VeG/MquPr1VRRW1nH/mSM4f0KKBAwhOtjRgoaz4Y7W2q2UyuqOAQP2zZ6Snkbn9N6qXP68YAPRIQG8c/VkRktKSgi/OFrQGKWUqvTeV0CQ97ECtNY63Ket60AyEN45udwe7v1kM6/8tIsp/WJ44vwxxIS2pbiyEKItjhg0tNY9Zs/MxvSUrNPoNKocTq5/Yw3f/VrElcekcfvJg6UgoRB+1uwpt91dQ1l0yZF3Dvnldi57+Re2FVbzr7NGcP6E9tnSVwjRNhI0vGSr185jQ24Fl7/yC/Z6Ny9fOp5jBsj2rUJ0FhI0vGQDps7hq017uPGttUSHBPDaNRMZlNCt6mIK0eVJ0PByuB0ynuFnr/6UzT0LNzEiKYLn52QQHyb/fQjR2XTaUUWl1MlKqUyl1Hal1B2+vl6tq1Z6Gn5UXF3H3z/ezLED43h77mQJGEJ0Up0yaCilzMBTwCxgKHC+UmqoL69pd8pWr/702YYC3B7NHbMGExTQYybtCdHldMqgAUwAtmutd2qt64G3gNm+vKDDLQPh/vTR2nwG9QpjcEK3WfojRLfUWYNGEpCz3+Nc73ONlFJzlVIrlVIri4qK2nxBu0t6Gv6SU1rLql1lnD5aNlISorPrrEGjqcUS+oAHWs/TWmdorTPi4to+JVOm3PrPx+vzAThddt8TotPrrEEjF0jZ73EykO/LC8qUW/9ZuDafsamRpERLWXohOrvOGjR+AQYopdKUUgHAecBCX15Qehr+kbmniq17qpg9OunoJwsh/K5TrtPQWru8mz19CZiBF7XWm3x1PY/2yEC4nyxcl4fZpDhlhOy+J0RX0CmDBoDW+jPgs464luyl4R9aaxauy2dKvxjiwqRyrRBdQWdNT3Uo2bXPP9bklJNTapfUlBBdiAQN9ttLQ8qIdKiFa/MJsJj4zbBe/m6KEKKZJGiwLz0VZJWeRkdxuT18sj6fE4bEE2az+rs5QohmkqDBvp5GsEWmfHaUn3aWUFxdL2szhOhiJGhglBABSU91pI/W5hMWaGHGoHh/N0UI0QISNJCB8I7mcLr5cuMeTh6egM0qxQmF6EokaGCURQeZcttRlmQWUlXnklpTQnRBEjTYbyBcehod4qO1+cSGBjI5PcbfTRFCtJAEDSQ91ZE25VeweGshp45MxGKW//kJ0dV02hXhHUl6Gr7ncLp5fPE25i3dSVRwABdN7uPvJgnRdbnqIPcX2LkEUibCgBM77NISNNhvcZ+MafjE8p0l3PnBBrKKa/htRjJ3nTKEyOAAfzdLiPZRtQfKdkFlLlTmQ0UeVOYZ901mCO8N4UkQkey9nwyWAKgsOOg9ueCohKg+EN0PYvpBTH/jfnAM7FkPWd8ZgWLXT+CygzLB9D9J0OhoDpeDQHMgJiXpkvZUYXfywOdbeHNFDqnRwcy/YiJT+8f6u1lCtF59DeSvgdyVkLcScldB1UG7NlhDICLJCBAeNxSsh8zPwZvROIQyQViicX5wNOzZCFs/BY9r3zkmK3icxv24wTD2YkifAX2ngi3CF3/pYUnQwJg9Jamp9rVsWxF/fGcdxdV1XDU9nZtOGCh7f4v2V18DhVuh+FeITIWksdDWyg6ueqjIgbIsowdRlm0cJTugaAtoj3FeVF/oMwWSxkHsAKM3Ed7b+BJXB+0jpzXUlnp7IHlGAAlPMo7QXmA+6KvY7YTy3cY1S3dARS4kjIC0YyHcvxWhJWggW722J7dH88Q323h88TYGxIfywpzxjEju2F9Copuyl8OuH2HvRu+xyfhS3X9TT5MVEkdB6iQj158yEYIiwVGx31Fu3NaUQPUeqN4LVXuN+1V7oabowM80B0BkHyNIDD4FkscbgSKkBb1mpSAkxjgSRx79fLPVm57q1/xrdBAJGsgGTO2lpLqOm95ey7JtxZw1Nol/njFCehfiUFobg7gb3jV+fSeONL7oE0Ya6ZkGHo+Rx9/+NWxfDDkrQLsBBdFp0GsYjDjXuI0daASQnJ+NY8Vz8NOTR2+LMkNovHGE9YbeY4xUUVRf44jsYzw2Seq6gQQNZKvX9rBqVxnXv7Gakpp6/nXWCM4bn4I6uIsueraSHbD+HVj/tpH6sdggOBY2vrfvnIhUI4gEhMCOb6Gm0Hg+cRRMuxn6zzSCS2DooZ8fN8joCYCRYipYB7krwGk3Uka2SO+t9wiONgaYTfLDpiUkaGDUnpK6U62jtealH7K5/7MtJEba+OCaKQxPknRUl+WqN37d56yA0p1G7t1db0zxdNcbj7UHgqKN9ExInPHFGxJrPOdxgbPW+KKurzFu6yph21dG7wIFadONGT9DTgNbuNHbKFh34OGogPRjof+J0O94CGth+XxLAKSMNw7RriRoAHannYgOnoHQXTzyVSZPfbuDE4f24pFzRxERJGXOu5TKAuPXeM4K40s9fy2464zXbJHGoLI5ACyBxmEONPLz5TlQW2x8uTdH/DA48R8w/BxjZtH+gqOh33HGITo9vwQNpdTDwGlAPbADuFRrXe597U7gcsAN3KC1/tLX7bG77CSYE3x9mW5nR1E1z363kzPHJPHob0dJOspf6qqg6FdjZk9ZtpHyiUzdd9jCjfM8bijcDLuXG0EiZ7kxQweMYNB7NEy4ElImQPKE5s3ScdVDbYkRQGpLjIHogGBj2mlAMFgbDunJdxf+6ml8DdyptXYppR4E7gRuV0oNBc4DhgG9gUVKqYFaa7cvG+Nwy0B4a9z3yWaCrGbuOmWIBIy28HiMX+/N+Tesr4WspbDre2OqadFWY3poI8UBM3/A6DGEJxkBor7KeC40AVInwsRrjCCRMNJI6bSUJcAILn6eBio6jl+Chtb6q/0eLgfO8d6fDbylta4DspRS24EJwE++bI9MuW25b7cW8m1mEX8+ZQhxYYH+bk7XoDVUFRi/9vduhsItxv2iTCMNlDQOkjMgKcNYb9Awk6gsG379CrZ9CVnLjPSRORDiBhpTS+MugfghxqKvyD7GlNLyXUaQaDgqcqHP5H3TUCNTmxekhDhIZxjTuAx423s/CSOINMj1PncIpdRcYC5Aampqmxogs6dapt7l4d5PNpMeG8KcKX393ZzOpWQHLP+fMde/rmrf4ag08v/Omn3nhiYYX/YZl0FdhbG6ePsiGnsK0f2MmT3Fv3ofpxvnDjwJ+kw1xhiaEhJrHEnjfPqnip7JZ0FDKbUIaGqg4M9a64+85/wZcAHzG97WxPm6iefQWs8D5gFkZGQ0eU5zaK1lnUYLvfJjNjuLa3jpkvEEWGT+OmCMFyx/Gr65z/gFH5ECgWHGEd7bez/c+OKPHwLxQw9ck9DAUWmUqWgoUeGug3GXwsDfdMqFXqLn8VnQ0FqfcKTXlVJzgFOBmVrrhi/9XCBlv9OSgfyD39ue6tx1aLSkp5qpqKqO/y7exnGD4jhusGzVChiD0B9dZ8xCGjgLTn2s9Tl+W7gx1TT92PZtoxDtxF+zp04GbgeO1VrX7vfSQuANpdSjGAPhA4AVvmyL7KXRMo98mYnd6ebuU4f6uyn+53bBT0/At/8yZgqd9ZyxQlnGCkQ35q8xjSeBQOBr76yb5Vrrq7XWm5RS7wCbMdJW1/l65pQEjebbkFvBO6tyuGJaGulxTazI7apc9caMJEf5vpRSYPi++2AsQLOXeqeXlhiPNy2A/NXGIrVT/t3yBWhCdEH+mj3V/wiv/RP4Z0e1RTZgah6tNX/7eBMxIQH8YeYAfzen7TweY53C+ndg84dgL2v5Z4T2gnNegmFnSu9C9BidYfaUXzVuwCRlRI5o4bp8Vu0q48GzRxBu68Krvgu3wLq3YOP7xvoGazAMOsVIK0Wne2c7VRw480lro1RGcIy3XpG3ZlFAqAQL0eNI0GhIT7W1Bn83pbXmnZU53LNwEyOTIzh3XMrR39QZOe2w6O/w89NGZdN+x8PMvxoBo6nid0KIJknQ6KFjGnsrHcSHBR5xJXd1nYs/L9jAR2vzmdY/lsd+NxqTqQv+ss5dCQuuhpJtMP5KmHFHy/ZCEEI06vFBw+E2xjR6UnrqwzV53PT2WtJjQ7hgYipnj00mKuTAEhKb8iu4/o017Cqp4daTBnLNjP6YO2PA0PrwKSJXPXz3AHz/mLFXwkUfSlE8IdqoxweNhp5GsCXYzy3pGNsLq7hrwQaGJ4UTYDZx36dbeOjLTE4dkcjvJ6UyNjWK15fv4t5PtxAVbOXNKycxMT3G380+VM4KWPwPY8OdmP7GYrn4IcaGPPFDjEVyH15j7PA2+kI4+f4O30tZiO5IgobTOxDeAxb31da7uHb+aoKsZp6/eDwJETY251fyxopdfLgmnw/W5BEfFkhhVR0zBsXx73NHERPayepK7d0Ei++FXz+HkHhjtXT5LmNh3f6b+YDx+nlv7tuYRwjRZj0+aDSkp3rCmMZfP9rEtsJqXrl0AgkRRpAc2juc+84YwZ2zhrBwXT6fbSjgymPSuXxaWucavyjNgm/vN7YIDQw3BrEnXm3s8NbAUWkU/yvcZKylGHuJsSezEKLd9PigUesyFqR3957GuytzeG9VLn84vj/TB8Yd8npIoIXzJ6Ry/oS2FX9sE62N9RL7V2ct321sDbrjG2Ovhqk3GkdTdZts4bJbmxA+1uODhsPlwGqyYjF133+KzD1V3P3RRialR3PTCQP93ZxDFW6BtfNh/btQvefA1wLCIKqPUd112i2yb4MQftZ9vymbqTuURXe5PZTbncQ2Mf5QU+fi2vmrCA208t/zxnSeGVC1pcYCu7VvGKU4TBYYcBL0nXbQrnORsoBOiE6kxwcNh8vRpVNT2wurueq1lewoqiEuLJDhvcMZnhTBsN7hDOsdwb+/yiSruIbXL59IfHgH/p0lO2DLQmOnOXedMf214ba22Eg3ueuh1wj4zb+MFdmhh6bNhBCdS48PGnaXvctOt/18QwG3vrsOm9XMrScNJKu4lk35FSzdVozbs2+LkZtPGMiU/h20mK1kByx9BNa/DdoNKGOzIHOgsTWoOdDYpS7jchh9ASSO7Jh2CSHahQSNLrjVq8vt4eEvM3l26U5GpUTy9O/H0jtyX4rN4XSzdU8VG/MqqK13cfm0dN83av9gYQ6ASdfAlD8YRf0kvSREt9Hjg0ZX27WvuLqOG95cw487Svj9xFT+etpQAi3mA86xWc2MTolkdEqk7xvUZLC4QcqEC9FN9figYXfZCQ3oGgXrfsku5YY311BSU8/D54zk3Aw/Fg+UYCFEjyRBw20n1tx5i9fV1rv4ZF0B81fsZl1OOclRQXxwzRSGJ/mgJIbHDXmrjDGHmAFgbSJtd3CwmHi1sW5CgoUQPYIEDae9U5ZFz9xTxRs/7+KDNXlUOVwMiA/lntOGcva45Pbdz0JrKFgLG94zpsBWFRjPKxNEpRl1nOIGQewg2LnEGyys0rMQoofq8UHD4Xb4pcKt1prcMjv55Xb2VtWxt8LBnkrj2FVSw8a8SgLMJk4ZkcAFE/swvm/UEcuYt/DiULrTCBLr3zFKhpusxjqJ4WcZAaNoq7HorigTMj83ZkJZbNKzEKKH82vQUErdCjwMxGmti5Xxrfg4cApQC1yitV7tyzZ05OI+rTUb8ir4fOMevti4h6zimgNeD7KaSYiwkRBu465TBnPOuBSiDypZ3oqLQlk2FKw78KgtNl7vMw2mXA9DTm+6NAcYaytKdxgFAKWWkxA9mt+ChlIqBTgR2L3f07OAAd5jIvC099YntNY+Dxpaa1btKmsMFHnldswmxeT0GC6Z0pe02BASImz0CrcRbrO0T29Ca6Nk+KpXIPNTcFQYz6CLON8AAAwwSURBVJssEDcEBp4MvUfDoFkQkXz0z7MEGGkqIUSP58+exmPAbcBH+z03G3hVa62B5UqpSKVUota6wBcNcHqceLTHZ0GjtKaem95ey9Jfiwgwm5g2IJYbTxjAiUN6HbLpUbuoLTX2v179ipFeCgiDobONAn6Jo4w9JyydrNS5EKJL8UvQUEqdDuRprdcd9Ms6CcjZ73Gu97lDgoZSai4wFyA1tXWVWX251euqXaVc/8YaSqrrufvUoZyb0c4D2A20ht0/wcoXYfNCo1RHUgac/iQMO1P2vxZCtCufBQ2l1CIgoYmX/gzcBZzU1NuaeE438Rxa63nAPICMjIwmzzmahqDRnivCtda88H0WD3y+lcRIGx9c66PpsU6HMZD98zOwZz0ERsC4OTB2DiQMb//rCSEEPgwaWusTmnpeKTUCSAMaehnJwGql1ASMnsX+K9aSgXxftbG9exoVdie3vbeOLzft5aShvXj43FFEBLVz76IyH355AVa9ZGw0FDcETv0PjPwdBHTNGlpCiK6jw9NTWusNQHzDY6VUNpDhnT21ELheKfUWxgB4ha/GM6D9ehpOt4efdpTwlw83kl9u5y//N4TLp6W13xRZgLzV8NNTsPlDYxHeoFNg4lWQNl1qOwkhOkxnW6fxGcZ02+0YU24v9eXFHK7Wb/VaXlvPkswiFm3Zy3e/FlHlcJEQbuPtqyYxrs9hpq62lMcD276EH5+EXd8bA9sT/r+9Ow+SorzDOP59drkWcIEFheVQIBjEECNIFKJY4A05jEd5lClMTFIeVCXRpAzEpKKmUimjFS2NFUl5H0ETtdRgVCwgJikVBUVAWQTEA0NYUJEjqBxv/njflXazsL0wOrszz6eqa7rf7pnpX7+z89t++523z4fDvw81gwrzHmZmLVD0pBFCGJiZD8Dkz+q9W9I8FUJgWf0m5tTVM2tJPfPeeJcdAXp17ciE4X045qDeHP35XnTuUIBDuvUDWHhvTBbvLIPq/nDCr2HkpHhLUzOzIil60iimZS8/DcD0J1/ixQO6M7TPPgzts8/Hd8D7YOt2nlnxDrPr6pldV8/b62OSGVZbzeTxQzh2WG8O6deNipbeDW/7ttjjaeVTsHltvC/2lvU7HzevhW1bYjfZ026J3WYrP4WeV2ZmLVTWSaNTx14cseUjTl17NecvquJ9YvfUnl060L+mM3WrN/Dhth1Uta/kyCG9mDx+COMP2pfabntw4Xzbh/DaU/Fudkv/Fi9iqzL+CruqR5yq+8bfUnSuidcsBh7l6xVm1qootgi1baNGjQrz5s3bsyev/Cfh7lPZ1mcE88beypJ1W3l1zUZWrtvMsNpqxh+0H0cMqqFT+8rmXwvikBsbV8deThvejtPql+DVmfDRRuhYHcd4GvZ1GHKcf0dhZkUjaX4IYVRLnlPWZxoADBqLTplG+/vPY8yCnzLmjDuhImeC+HATrHoe3nwW3nw6Du63qZ7/+2lJl31h+ClxfKdBR/tX2WbWZjlpQBzZddMaeHwKPHYpTLym6WahHdth+Sx4bU68JrF6YRz9VRXQezgceDx0GxCbmar7pamvL16bWclw0mgw+sLYlPT0DfGLfuyPd67buAZeuBPm3w4bVsUhwvt/GcZeAvuPifNODGZWBpw0so67MiaIWVdC1z7QfUD89XXdDNixDQaPg5N+E0eJbfcpDDhoZtbKOWlkVVTAyTfC5np4+KJYVtUj3nho1HnQ83PF3T8zsyJz0misXQc44y6Y/SvoOxK+8M14z2wzM3PSaFKnaph4dbH3wsys1ako9g6YmVnb4aRhZma5OWmYmVluThpmZpabk4aZmeXmpGFmZrk5aZiZWW5OGmZmlltJ3E9D0lrgjT18ei9gXQF3py0p19gdd3lx3Lt2QAhh35a8aEkkjb0haV5Lb0JSKso1dsddXhx3Ybl5yszMcnPSMDOz3Jw04I/F3oEiKtfYHXd5cdwFVPbXNMzMLD+faZiZWW5lnTQknSRpqaTlkqYUe3/2lqQBkuZIWiLpZUk/TOU1kp6UtCw99kjlknR9in+hpJGZ1zo3bb9M0rnFiqklJFVKelHSjLQ8SNLcFMN9kjqk8o5peXlaPzDzGlNT+VJJJxYnkvwkdZd0v6S6VO9jyqG+JV2cPuOLJU2X1KkU61vSrZLqJS3OlBWsfiUdJmlRes71ktTsToUQynICKoEVwGCgA/AScHCx92svY6oFRqb5fYBXgYOB3wJTUvkU4Ko0PxF4DBAwGpibymuA19JjjzTfo9jx5Yj/EuBPwIy0/GfgrDR/E3Bhmr8IuCnNnwXcl+YPTp+DjsCg9PmoLHZczcR8B/C9NN8B6F7q9Q30A1YCVZl6/nYp1jdwNDASWJwpK1j9As8BY9JzHgMmNLtPxT4oRayMMcATmeWpwNRi71eBY3wYOB5YCtSmslpgaZqfBpyd2X5pWn82MC1T/ontWuME9AdmAccAM9IfwTqgXeP6Bp4AxqT5dmk7Nf4MZLdrjRNQnb481ai8pOs7JY230pdgu1TfJ5ZqfQMDGyWNgtRvWleXKf/Edruayrl5quGD12BVKisJ6RR8BDAX6B1CWA2QHvdLm+3qGLTFY3MdcCmwIy33BNaHELal5WwMH8eX1r+ftm9rcQ8G1gK3pWa5myV1ocTrO4TwNnAN8Cawmlh/8yn9+m5QqPrtl+Ybl+9WOSeNptruSqIrmaSuwAPAj0IIG3a3aRNlYTflrZKkrwH1IYT52eImNg3NrGtTcRP/ax4J/CGEMALYTGyu2JWSiDu14Z9MbFLqC3QBJjSxaanVd3NaGucexV/OSWMVMCCz3B/4d5H2pWAktScmjHtCCA+m4jWSatP6WqA+le/qGLS1Y3Mk8A1JrwP3EpuorgO6S2qXtsnG8HF8aX034F3aXtyrgFUhhLlp+X5iEin1+j4OWBlCWBtC2Ao8CHyF0q/vBoWq31VpvnH5bpVz0ngeODD1uOhAvED2SJH3aa+kng+3AEtCCL/LrHoEaOgxcS7xWkdD+aTU62I08H463X0COEFSj/Rf3QmprFUKIUwNIfQPIQwk1uPsEMI5wBzg9LRZ47gbjsfpafuQys9KvW0GAQcSLxS2SiGE/wBvSRqaio4FXqHE65vYLDVaUuf0mW+Iu6TrO6Mg9ZvWbZQ0Oh3HSZnX2rViX+Qp8gWmicQeRiuAy4q9PwWI5yji6eVCYEGaJhLbb2cBy9JjTdpewI0p/kXAqMxrnQcsT9N3ih1bC47BOHb2nhpM/BJYDvwF6JjKO6Xl5Wn94MzzL0vHYyk5epIUewIOBealOn+I2Dum5OsbuAKoAxYDdxF7QJVcfQPTiddtthLPDL5byPoFRqVjuAL4PY06VTQ1+RfhZmaWWzk3T5mZWQs5aZiZWW5OGmZmlpuThpmZ5eakYWZmuTlpmCWStktakJl2O/KxpAskTSrA+74uqdfevo7ZZ8Fdbs0SSZtCCF2L8L6vE/vUr/us39uspXymYdaMdCZwlaTn0jQklV8u6Sdp/geSXkn3Mbg3ldVIeiiVPSvpkFTeU9LMNMjgNDJjAEn6VnqPBZKmKd4jpFLS7Yr3jlgk6eIiHAYzwEnDLKuqUfPUmZl1G0IIhxN/NXtdE8+dAowIIRwCXJDKrgBeTGU/A+5M5b8E/hXiIIOPAPsDSBoGnAkcGUI4FNgOnEP81Xe/EMLwEMIXgdsKGLNZi7RrfhOzsrElfVk3ZXrm8dom1i8E7pH0EHE4D4jDupwGEEKYnc4wuhFvrHNqKn9U0ntp+2OBw4Dn0w3UqoiD0f0VGCzpBuBRYOaeh2i2d3ymYZZP2MV8g68Sx/05DJifRlPd3dDTTb2GgDtCCIemaWgI4fIQwnvAl4C/A5OBm/cwBrO95qRhls+ZmcdnsiskVQADQghziDeC6g50Bf5BbF5C0jhgXYj3N8mWTyAOMghx8LnTJe2X1tVIOiD1rKoIITwA/II4/LlZUbh5ymynKkkLMsuPhxAaut12lDSX+I/W2Y2eVwncnZqeBFwbQlgv6XLiXfUWAv9l53DWVwDTJb0APEUc6psQwiuSfg7MTIloK/HMYkt6nYZ/8qYWLmSzlnGXW7NmuEus2U5unjIzs9x8pmFmZrn5TMPMzHJz0jAzs9ycNMzMLDcnDTMzy81Jw8zMcnPSMDOz3P4HJARBDWaLFAoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label = 'avg')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label = 'min')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label = 'max')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Rewards')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Rendering Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "done = False\n",
    "state = env.reset()\n",
    "while not done:\n",
    "    action_indx = np.argmax(q_table[get_discrete_state(state)])\n",
    "    next_state, _, done, _ = env.step(action_space[action_indx])\n",
    "    state = next_state\n",
    "    env.render()\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (spinningup)",
   "language": "python",
   "name": "spinningup"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
